Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement workflow for automatic ECS templates generation #586

Open
wants to merge 35 commits into
base: master
Choose a base branch
from
Open

Implement workflow for automatic ECS templates generation #586

wants to merge 35 commits into from

Conversation

QU3B1M
Copy link
Member

@QU3B1M QU3B1M commented Dec 10, 2024 •
edited
Loading

Description

Implement a Docker image that generates ECS index templates based on the mappings definitions, this image is consumed by a Docker Compose that sets all the required env variables and configures the volume where the ECS mappings are stored. A bash utility was generated to handle the execution of the docker compose.

Add another bash tool that launches the ECS mapping generation utility when a modification on a ECS definition is detected, automatically copying the resulting template to wazuh-indexer-plugins repository generating a new branch and a PR with the changes.

Related Issues

Resolves #540

Working evidence

Mapping generator

bash docker/ecs/mapping-generator.sh run alerts     
[+] Running 1/0
 ✔ Container wazuh-ecs-generator  Created                                                                                                                                                           0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped

GitHub Action ECS utility

Since it requires an GitHub access token to generate the PR, the GHA Workflow was validated on a fork of this repository

GHA bash utility local execution 
GITHUB_TOKEN=<GITHUB_TOKEN> bash ecs/scripts/generate-pr-to-plugins.sh  -b ci/540-ecs-index-automatic-generation

---> Fetching and extracting modified ECS modules...
Updated ECS modules: alerts generate.sh generator scripts states-inventory-hardware
Relevant ECS modules: alerts states-inventory-hardware

---> Running ECS Generator script...
[+] Running 2/0
 ✔ Network generator_default      Created                                                                                                                      0.0s 
 ✔ Container wazuh-ecs-generator  Created                                                                                                                      0.0s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/alerts/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/alerts/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                                      0.0s 
Processed ECS module: alerts
[+] Running 2/0
 ✔ Container wazuh-ecs-generator  Removed                                                                                                                      0.0s 
 ✔ Network generator_default      Removed                                                                                                                      0.0s 
[+] Running 2/0
 ✔ Network generator_default      Created                                                                                                                      0.0s 
 ✔ Container wazuh-ecs-generator  Created                                                                                                                      0.1s 
Attaching to wazuh-ecs-generator
wazuh-ecs-generator  | Loading schemas from git ref v8.11.0
wazuh-ecs-generator  | Running generator. ECS version 8.11.0
wazuh-ecs-generator  | Loading user defined schemas: ['/source/ecs/states-inventory-hardware/fields/custom/']
wazuh-ecs-generator  | Replacing unsupported types in generated mappings
wazuh-ecs-generator  | Deleting the "tags" field from the index template
wazuh-ecs-generator  | Removing multi-fields from the index template
wazuh-ecs-generator  | Mappings saved to /source/ecs/states-inventory-hardware/mappings/v8.11.0
wazuh-ecs-generator exited with code 0
[+] Stopping 1/0
 ✔ Container wazuh-ecs-generator  Stopped                                                                                                                      0.0s 
Processed ECS module: states-inventory-hardware
[+] Running 2/0
 ✔ Container wazuh-ecs-generator  Removed                                                                                                                      0.0s 
 ✔ Network generator_default      Removed                                                                                                                      0.1s 

---> Cloning QU3B1M/wazuh-indexer-plugins repository...
Cloning into '/Users/quebim_wz/Wazuh/forked/wazuh-indexer/../wazuh-indexer-plugins'...
remote: Enumerating objects: 1280, done.
remote: Counting objects: 100% (356/356), done.
remote: Compressing objects: 100% (214/214), done.
remote: Total 1280 (delta 258), reused 142 (delta 142), pack-reused 924 (from 2)
Receiving objects: 100% (1280/1280), 426.50 KiB | 2.04 MiB/s, done.
Resolving deltas: 100% (449/449), done.
Already up to date.
✓ Set QU3B1M/wazuh-indexer-plugins as the default repository for the current directory

---> Committing and pushing changes to QU3B1M/wazuh-indexer-plugins repository...
branch 'ci/540-ecs-index-automatic-generation' set up to track 'origin/ci/540-ecs-index-automatic-generation'.
Switched to a new branch 'ci/540-ecs-index-automatic-generation'
From https://github.com/QU3B1M/wazuh-indexer-plugins
 * branch            ci/540-ecs-index-automatic-generation -> FETCH_HEAD
Already up to date.
Copying ECS templates to the plugins repository...
  - Copy template for module 'alerts' to 'index-template-alerts.json'
  - Copy template for module 'states-inventory-hardware' to 'index-template-hardware.json'
Nothing to commit, working tree clean.

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@QU3B1M QU3B1M self-assigned this Dec 10, 2024
@QU3B1M QU3B1M changed the title Add initial ecs Dockerfile Implement workflow for automatic ECS templates generation Dec 10, 2024
QU3B1M added 16 commits December 10, 2024 12:33
@QU3B1M
Add basic entrypoint script
318117a
@QU3B1M
Add dependencies installation to Dockerfile
b1fd175
@QU3B1M
Remove unused variables
b47243d 
Rename local variables to lower case
@QU3B1M
Remove upload related code on generate.sh tool
25d16d3
@QU3B1M
Fix compatibility issues with latest python version
ba80cb5 
Update generate.sh script to use env variables
@QU3B1M
Remove unnecesary script
f49adbd
@QU3B1M
Rename ECS source directory
38972b1
@QU3B1M
Implement docker compose to ecs generator tool
ef1dd31
@QU3B1M
Working docker compose file for ECS generator
8827723
@QU3B1M
Add doscstrings to Dockerfile
76fade9
@QU3B1M
Add script to execute the ECS generator tool
368315d
@QU3B1M
Update ecs generator launcher script
c25a034
@QU3B1M
Mitigate warnings on container down and stop actions
118cdbe 
Stop container after each execution
@QU3B1M
Rename ecs util to mapping-generator
c4d323c
@QU3B1M
Implement generate-ecs-mapping GHA workflow to auto-generate PRs with…
c7f4c1b 
… the updated template
@QU3B1M
Update GHA plugins version
60cf765
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from 44378cc to 60cf765 Compare December 13, 2024 18:32
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from 15836e7 to bad34a6 Compare December 13, 2024 18:54
@QU3B1M QU3B1M marked this pull request as ready for review December 13, 2024 18:57
@QU3B1M QU3B1M requested a review from a team as a code owner December 13, 2024 18:57
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from 883db15 to c3b99d1 Compare December 13, 2024 20:03
@QU3B1M
Fix generator.sh path on Dockerfile and update GHA to work with multi…
46e9135 
…ple modules at once

Filter out dirs that dont correspond to any template
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from c3b99d1 to 46e9135 Compare December 13, 2024 20:32
AlexRuiz7
AlexRuiz7 requested changes Dec 23, 2024
View reviewed changes
.github/workflows/generate-ecs-mappings.yml Outdated Show resolved Hide resolved
.github/workflows/generate-ecs-mappings.yml Outdated Show resolved Hide resolved
docker/ecs/images/generator.sh Outdated Show resolved Hide resolved
docker/ecs/images/generator.sh Outdated Show resolved Hide resolved
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from 29b5734 to da1a00f Compare December 26, 2024 11:31
@f-galland
Copy link
Member

I tested this in a personal fork:

The Workflow was failing due to a non-zero exit code in the generate-pr-to-plugins.sh script triggering set -e to exit altogether.

Fixing that got the PR generation working:

@f-galland
Copy link
Member

The latest changes make updates to mappings trigger updates in PR description work.

@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch 5 times, most recently from 2bc22d8 to 085bdb6 Compare January 6, 2025 16:42
@QU3B1M
Update GHA to use the new tool
663aab8 
Update ecs README with new tool usage

Update ECS generator script name

Remove unused index upload option

Fix issue with exit codes on branch checking
@QU3B1M QU3B1M force-pushed the ci/540-ecs-index-automatic-generation branch from 085bdb6 to 663aab8 Compare January 6, 2025 16:45
AlexRuiz7
AlexRuiz7 requested changes Jan 7, 2025
View reviewed changes
.github/workflows/generate-ecs-mappings.yml Outdated Show resolved Hide resolved
.github/workflows/generate-ecs-mappings.yml Show resolved Hide resolved
ecs/generator/images/Dockerfile Outdated Show resolved Hide resolved
ecs/README.md Outdated Show resolved Hide resolved
ecs/generator/mapping-generator.yml Outdated Show resolved Hide resolved
ecs/scripts/generate-pr-to-plugins.sh Outdated Show resolved Hide resolved
ecs/scripts/generate-pr-to-plugins.sh Outdated Show resolved Hide resolved
ecs/scripts/generate-pr-to-plugins.sh Show resolved Hide resolved
ecs/README.md Outdated Show resolved Hide resolved
QU3B1M added 2 commits January 7, 2025 13:04
@QU3B1M
Remove hardcoded ECS version
67a71bd 
Update GHA workflow to only work on official wazuh-indexer repo

Add PR generation tool description at main ECS README
@QU3B1M
Add docstrings to functions
73a0b1d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AlexRuiz7 AlexRuiz7 AlexRuiz7 approved these changes

Assignees

@QU3B1M QU3B1M

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Workflow for automatic generation of index templates on changes
3 participants
@QU3B1M @f-galland @AlexRuiz7